import pandas as pd
import numpy as np
import statsmodels.api as sm
from matplotlib import pyplot as plt
from plotnine import *
from great_tables import GT, md
from mizani.formatters import percent_format
from scipy.stats import chi2
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay, roc_curve, roc_auc_scoreEvaluating Logistic Regresion Outcomes
Evaluating Performance: Statistical Tests (continued)
- To the purposes of our analysis, a constrained case, as a baseline, can be the case where the \(Y\) has no predictors (i.e, all \(\beta\)’s are equal to zero)
Confusion Matrix
- Diagonal cells indicate the True Positive and True Negative cases
- Off-Diagonal cells indicate the False Positive and False Negative cases
- Type I Errors (\(2,477\)) are the False Positive cases: we wrongly classified customers that did not churn as churned!
- Type II Errors (\(650\)) are the False Negative cases: we wrongly classified customers that did churn as not churned!
Confusion Matrix Implementation
<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay object at 0x000002AE9F7C6BA0>
# Predict churn using the trained logistic regression model
predicted_churn = (
(reg.predict() >= y.mean()) # Assuming a threshold of 0.5 for classification
.astype(int)
)
#Estimate the Confusion Matrix
CM=confusion_matrix(y,predicted_churn)
#Chart
ConfusionMatrixDisplay.from_predictions(y, predicted_churn)
#Title
plt.title('Confusion Matrix')
#Show
plt.show()Putting all together
- As we increase our threshold, \(p^\star\), we minimize Type II error (i.e, we identify the churned customers) as we’re identifying the true positives
- At the same time, however, we are increasing the Type I error, since there is going to be a higher number of false positives!
Finding the optimal \(p^{\star}\)
- If, for example, we want to find \(p^{\star}\) such that it maximizes the sum of specificity + sensitivity, we can:
- Redo our confusion matrix for each \(p^{\star}=\{0,0.01,0.02,...,0.99,1\}\)
- Calculate the specificity and the sensitivity
- Pick the threshold that maximizes the sum of both metrics
- Note that, depending upon the problem, we might want to use a different criterion to find the optimal \(p^\star\)
# Predict churn probabilities
probs = reg.predict()
# Calculate specificity and sensitivity
fpr, tpr, thresholds = roc_curve(y, probs)
specificity = 1 - fpr
sensitivity = tpr
# Find the best threshold value
best_threshold_index = np.argmax(specificity + sensitivity)
best_threshold = thresholds[best_threshold_index]
#Create a pandas data.frame and plot it
result=pd.DataFrame({
'thresholds':thresholds,
'sp_se': specificity + sensitivity
})
Plot=(
ggplot(result, aes(x='thresholds',y='sp_se'))+
geom_point(size=0.5,color='darkorange')+
theme_minimal()+
geom_vline(xintercept=best_threshold,linetype='dashed')+
labs(x='Threshold',
y='Specificifity + Sensitivity',
title='Best value achieved around 0.22')+
theme(plot_title = element_text(size=15,face='bold'),
axis_text = element_text(size=12),
axis_title = element_text(size=12),
figure_size=(15,6))
)
Plot.show()Comparing among different classifiers
Say that, for some reason, you have ommitted
agefrom the analysis. How can you assess how the predictive power of your model is going to deteriorate?One way to do this is to analyze what we call Area under the Curve (or simply AUC):
AUC under different models
- A Receiver Operating Characteristic curve (or simply ROC curve) is a graph showing the performance of a classification model at all classification thresholds. This curve plots two parameters:
- The True Positive Rate (or TPR)
- The False Positive Rate (or FPR)
The AUC provides an aggregate measure of performance across all possible classification thresholds
However, such metric is not very applicable metric whenever the costs of Type I and Type II errors are very different
# Define independent and dependent variables for Reg_1 (without 'age')
X1 = X.drop('age',axis=1)
# Define independent and dependent variables for Reg_2 (with 'age')
X2 = X
# Dependent variable
y = data['churn']
# Fit logistic regression models
reg_1 = model = sm.Logit(y, X1.astype(float)).fit()
reg_2 = model = sm.Logit(y, X2.astype(float)).fit()
# Predict probabilities for both models
probs_1 = reg_1.predict()
probs_2 = reg_2.predict()
# Compute ROC curves
fpr_1, tpr_1, _ = roc_curve(y, probs_1)
fpr_2, tpr_2, _ = roc_curve(y, probs_2)
# Compute AUC scores
auc_1 = roc_auc_score(y, probs_1)
auc_2 = roc_auc_score(y, probs_2)
# Plot ROC curves
Data=pd.concat([pd.DataFrame({'Model':'Without Age','FPR':fpr_1,'TPR':tpr_1}),
pd.DataFrame({'Model':'With Age','FPR':fpr_2,'TPR':tpr_2})],axis=0)
Plot = (
ggplot(Data,aes(x='FPR',y='TPR',fill='Model'))+
geom_point(stroke=0)+
theme_minimal()+
annotate(geom='text',x=0.5,y=0.5,label=('Model 1 (without Age): ' + auc_1.round(2).astype('str')))+
annotate(geom='text',x=0.4,y=0.9,label=('Model 2 (with Age): ' + auc_2.round(2).astype('str')))+
labs(x='FPR (1 - Specificity)',
y='TPR (Sensitivity)',
title='Model with Age outperforms the other for any threshold!')+
theme(plot_title = element_text(size=12,face='bold'),
axis_text = element_text(size=10),
axis_title = element_text(size=12),
legend_position ='bottom',
figure_size=(15,6))
)
Plot.show()References
